Virtual reality systems are known, comprising a headset which, when placed over a user's eyes, creates and displays a three dimensional virtual environment in which a user feels immersed and with which the user can interact in a manner dependent on the application. For example, the virtual environment created may comprise a game zone, within which a user can play a game.
More recently, mixed reality systems have been developed, in which an image of a real world object can be captured, rendered and placed within a 3D virtual reality environment, such that it can be viewed and manipulated within that environment in the same way as virtual objects therein. Other mixed reality systems have also been developed that enable virtual images to be blended into a user's view of the real world. In order to make such systems effective, the blended images and the environment scene must be combined seamlessly so that an object aligns correctly with the scene into which it is being placed. Realistic merging of virtual and real objects into a scene also requires that the position and movement of objects is realistic, and that they behave in a physically plausible manner within the environment. Thus, for example, they may need to be occluded or shadowed by objects in the scene.
Theoretically, resolving occlusions amounts to a comparison of the depth of captured and virtual objects with the depth of the scene into which they are to be placed, and systems have been developed that employ 3D mapping and 3D reconstruction to resolve this issue. However, there are a number of problems associated with these known techniques. Firstly, the processing and hardware overhead for 3D reconstruction techniques is relatively high. Also, whatever method is used for 3D reconstruction, the accuracy or density of the recovered map is often not sufficient to produce a good estimate of the occluding boundaries, especially for complex scenes. Still further, camera motion between two frames is not always perfectly known in real augmented reality applications.
It would, therefore, be desirable to provide a passive approach to the depth comparison in mixed and augmented reality systems, and aspects of the present invention are intended to address at least some of these issues.